Forecasting, Model Averaging and Model Selection

Abstract This paper explores forecasting using model selection and model averaging and attempts to draw conclusion both in the context of stationarity and non-stationarity. Model averaging tends to be viewed as a polar opposite of model selection; often the motivation for averaging is to avoid the pitfalls of selecting models. However, selection cannot be avoided since every possible model cannot be averaged over, and nor would it be sensible to. In fact, despite bold claims about averaging as opposed to selection, the most popular model averaging algorithms incorporate quite judicious selection procedures to reduce the model pool. Furthermore, despite advances in the theory and practice of model selection for averaging (e.g. Hendry and Krolzig, 2005), similar progress for forecasting has been more difficult, owing primarily to the difficulties caused by structural break non-stationarity. In this paper we use simulation to assess Bayesian Model Averaging (BMA) and Autometrics Model Selection (AMS) as tools for forecasting. It is found that decisions about the retention of borderline-significant variables are costly for forecasting, and that this affects any forecast technique that incorporates selection. Despite very different selection procedures, in the stationary case the implications for BMA and AMS are very similar. This finding is somewhat altered in the case of structural breaks, and the paper discusses methods by which to improve forecasts in this context.